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Abstract: This paper examines the practice of participatory evaluation through an 
exploratory single case study of the Evaluation Team of Books & Beyond, a co- 
curricular sendee-learning program of the Global Village Living-Learning 
Center at Indiana University. The paper, which is authored by three 
undergraduate members of the evaluation team and their faculty advisor, 
juxtaposes the process of conducting the evaluation and reporting the results with 
reflections from the Evaluation Team participants on conducting youth 
participatory action research, which offers a means of improving youth-serving 
programs and developing a greater understanding of why youth choose to 
participate in these programs. In their review of the implementation of their 
evaluation project, the team noted that the difficulties of getting past lessons 
learned to methodological rigor in service-learning evaluation are compounded 
by the realities of engaging in a student-faculty partnership in a co-curricular 
service-learning context. 
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In 2010-11, undergraduate students from the Indiana University Global Village Living-Learning 
Center implemented a youth participatory evaluation of Books & Beyond, a student-led, 
collaborative, co-curricular service-learning project founded in 2008. Their purpose was to 
identify the areas where Books & Beyond was working well and other areas where the 
experiences of the participants could be improved. Upon completion of the evaluation, all 
evaluation team participants reflected on the experience of being involved with the evaluation of 
the project, including working closely with each other, and participatory evaluation itself. The 
study presented here centers on the reflections of the participants following the completion of the 
evaluation project and focuses on a single exploratory research question: What lessons about the 
process may be learned from conducting a youth participatory evaluation in the context of a 
student-led co-curricular service-learning project? 

We begin with descriptions of the IU Global Village Living-Learning Center and its 
Books & Beyond International Service Project. Theoretical foundations for the study, research 
design, and Evaluation Team reflections follow. While our purpose is not to report on the results 
of the evaluation itself, we will also provide a few highlights from the evaluation report and its 
impact. 


I. The Global Village Living-Learning Center. 
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The Global Village Living-Learning Center, where Smith, Stevenson and Ryan lived for part of 
their undergraduate studies, is one of several residential learning communities on the IU 
Bloomington campus. Sponsored by the IU College of Arts and Sciences, the Global Village 
attracts students who are interested in international studies and foreign languages, regardless of 
their majors. The focus on international interests is a departure from the traditional language 
house, where students share a residence for the purpose of practicing common foreign languages 
together. The Global Village also offers fonnal and informal learning opportunities throughout 
the academic year, including the introductory “Q” course taught by peer instructors, which 
invites students to consider the political, social, cultural and economic dimensions of 
globalization and their roles in the process. 

Living-learning centers such as the Global Village provide an academically enriched 
environment that makes residential halls more than simply places where students live and study. 
Through curricular and co-curricular activities on a shared theme or common interest, these 
learning communities integrate learning into the daily lives of the residents and mitigate the 
sense of disconnection between academics and ‘real life’ that undergraduates often experience. 
Academic living-learning communities have been shown to have positive effects on student 
satisfaction, retention and academic achievement (Forest & Kinser, 2002; Inkelas, 1999; Inkelas 
et al., 2006; Inkelas & Weisman, 2003) and have been recognized for their track record in 
promoting experiential learning and critical application of new skills (Brower & Inkelas, 2010). 

II. Books & Beyond International Service Project. 

Because the Global Village strives to create a close-knit, diverse community of learners, the 
director of the center and his staff partnered with IU alumni and faculty to create Books & 
Beyond in 2008. The project was developed as a co-curricular service-learning activity that 
would provide a focal point for resident activities, while developing leadership skills and 
enriching educational experience. Over time, Books & Beyond has developed into a multi-year, 
multi-partner collaboration between the Global Village and two partners: TEAM Charter 
Schools: A KIPP Region, located in Newark, NJ, and the Kabwende Primary School, located in 
Kinigi, Rwanda. The project partners work together to engage young students at TEAM and 
Kabwende Primary School in authoring, illustrating, publishing and marketing collections of 
short stories. Since 2008, the project has printed five volumes of The World is Our Home, a 
collection of stories written and illustrated by young authors. Each participant receives a copy of 
the book, and in Rwanda these books have provided much needed reading material for primary 
school students. Books & Beyond has received considerable local media attention and has won 
several awards for youth service and experiential learning. In 2013, the books will be available in 
digital format. 

Books & Beyond embodies the living-learning center philosophy by aiming to increase 
students’ civic engagement, promote student retention, and develop practical skills through 
tutoring and mentoring opportunities, leadership opportunities, cross-cultural communication, 
project management, public speaking and fund-raising. Because Books & Beyond is co- 
curricular, all of the students engaging in the process of developing the books receive no 
academic credit for their efforts. 

A. Project Organization. 
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Although living-learning center staff and university faculty played more active roles in the 
founding and development of Books & Beyond, the nature of the project called for ever- 
increasing student involvement and control in planning, managing, and evaluating the project 
each year. As we will demonstrate with the Evaluation Team, this desire to ensure that the project 
is truly student-led is taken very seriously by the project participants and their mentors. 

Numbers demonstrate how the project has thrived since beginning in 2008. Twenty-five 
Global Village students were involved with Books & Beyond in its inaugural year (2008-9). 
Twelve TEAM students participated in the project and contributed stories to the collection. In 

2009- 10, forty-six Global Village residents and twelve TEAM students joined in the project. In 

2010- 11, fifty-five Books & Beyond participants represented 33% of the total population, with 
18% rejoining the project for a second or third year, and fourteen TEAM students participated 
(see Figure 1). These trends in growth have continued in 2011-12 and 2012-13. Each year, a 
majority of the Indiana University participants are first-year students; approximately one-third of 
all Global Village residents have chosen to be involved in Books & Beyond. 
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Figure 1. U.S. Participants in Books & Beyond (2010-11). 
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Figure 2. U.S. Participant Roles in Books & Beyond (2010-11). 

Participants can volunteer with Books & Beyond in several ways. The TEAM students 
mainly work as writers and illustrators. Indiana University students who elect to be Writing 
Partners are paired up with TEAM students to work on crafting their stories. This role is the most 
common, as 44% of the Indiana University participants in the project are Writing Partners (see 
Fig. 2). The second most common team, the Collaborators (20%), helps to ensure that the stories 
are linguistically and culturally relevant. For instance, they teach the writers to avoid mentioning 
aspects of their lives that might be unfamiliar to Rwandan students, such as playing video games 
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or going to a fast-food restaurant. The Collaborators also develop skills in fonnatting and book 
design while editing over thirty stories each year. Other students maintain pen pal relationships 
with teachers at Kabwende Primary School (7%). The Fundraisers (11%) write grants and raise 
approximately $20,000 each year. The Documenting Team (15%) creates short promotional 
videos that share Books & Beyond’s challenges and successes. The Evaluation Team (3%) 
evaluates the impact of the project and looks for ways to improve it. And the Leadership Team 
takes an active role in organizing all of the teams, with a leadership team member designated as 
the head of each individual team. In 2010-11, most participants contributed one or two hours per 
week (see Figure 3). 
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Figure 3. Hours Devoted Per Week to Books & Beyond by U.S. Participants (2010-11). 

B. Project Timeline. 

Several major events characterize Books & Beyond’s annual activities (see Table 1). Work on 
the book begins in the early fall and is concluded in late April, when the book is sent to the 
printer. In October, the TEAM students visit IU, tour the campus, meet with their Global Village 
writing partner, and begin working on their stories together. Between October and January, the 
writing partners converse three times over Skype. They also use social networking such as 
Facebook, Twitter, e-mailing, and text messaging to keep up informally. In January, the Global 
Village students go to Newark over the Martin Luther King, Jr. Day weekend to hold work 
sessions so the writing partners can complete their stories together. In the summer, students from 
the Global Village and TEAM travel to Rwanda to distribute the books to the students at 
Kabwende Primary School and to help the Rwandan students work on their stories for the next 
year’s collection. As of 2012, Books & Beyond has produced four volumes of stories and 
distributed over 8,000 copies to students in Rwanda. In the second, third, and fourth years, a total 
of sixty Rwandan students contributed their stories to the book. 

TIT. Youth Participatory Evaluation in Co-curricular Service Learning. 

Adding a youth-led evaluation team in the second year fit well with the goal of increasing 
student leadership in Books & Beyond. We worked from a Vygotskian (1978) theoretical 
foundation that posits that learning through action research can be developmental and 
participatory. Because the approach recognizes the expertise and leadership qualities of all 
participants, regardless of age or status, and helps to balance power relationships, the benefits of 
youth participatory evaluation can include transformation of ways of knowing (Checkoway, 
Dobbie, & Richards-Schuster, 2003). YPE helps to build social and civic competencies, self- 
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confidence and social capital, while encouraging identity exploration and fonnation (Sabo, 2003, 
2008) by allowing participants opportunities to enact roles that normally would not be within the 
realm of possibility in traditional classes or activities (Heath, 2000). As a form of participatory 
action research, YPE follows a reflective spiral in which participants collaboratively examine a 
question of community interest by following devising questions, collecting and interpreting data, 
and then reflecting on the process (McIntyre, 2008; Walter, 2009). Participants can be 
empowered when the results of their evaluation work are applied to their programs (Gong & 
Wright, 2008). Given this potential, youth organizers and researchers engaged in youth 
participatory research have opined that "not involving young people as researchers constitutes a 
missed opportunity for evaluation research" (Krenichyn et al, 2007, p. 603; London, 
Zimmerman, & Erbstein, 2003). Without engaging youth, a gap may exist between the adults 
performing the evaluation and the youth participants, who are acutely aware of the project and its 
impact on them and their communities. London, Zimmerman, and Erbstein (2003) have noted 
that “connecting youth, organizational, and community development can produce generative and 
self-sustaining processes that serve to address key social issues and revitalize communities and 
the organizations and individuals within them” (p. 35). 


Table 1. Timeline for Major Books & Beyond Activities 


Fall 

Semester 

Leadership team issues a call for participation. 

TEAM Charter Schools students visit IU to meet their writing mentors and begin 
working on their stories. 

Students continue writing with their mentors via Skype and phone. 

Spring 

Semester 

Students meet again, this time in Newark, NJ, to finish the stories. 

Editing and revision of stories for The World is Our Home. 

Book sent to the publisher for printing. 

Summer 

Students from the U.S. travel to Rwanda to visit Kabwende School and deliver the 
books. 

Rwandan students write their stories to be included in next year’s volume. 


Because it can encourage growth that occurs beyond the margins of traditional learning 
arrangements, participatory action research methodology fits well with co-curricular service 
learning. The co-curricular variety, which is less frequently studied, aspires to the same ideals as 
curricular service learning, but does not occur in credit-bearing courses. Co-curricular service 
learning provides opportunities for participants to work outside of the expected roles of 
undergraduates as learners or consumers of knowledge. As with service learning for credit¬ 
bearing courses, one of the major requirements of co-curricular service learning is the emphasis 
on giving equal weight to service as well as learning (Keen & Hall, 2009) and benefiting both the 
service providers and the recipients equally (Furco, 2003). This emphasis on both service and 
learning distinguishes curricular and co-curricular service learning from other forms of 
experiential learning such as volunteerism, internships, or community service, which may 
emphasize service or learning, but not both. 

Recent scholarship on teaching and learning (SoTL) values student-faculty partnerships 
in participatory research has emphasized the need for learning experiences that are both active 
and integrative. Recent research has shown that undergraduates valued learning opportunities 
that are characterized by experiential learning, significant interactions with faculty members, 
meaningful peer relationships, and scope for leadership and responsibility (Bowen et al., 2011). 
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Literature on training students for involvement in participatory evaluation is relatively 
sparse. We relied on an account of a similar student-faculty partnership by Tagor and Cuellar 
(2008), who described seven steps for designing and implementing a project: (1) Project 
planning and start-up; (2) Team-building (including critical thinking, skills development); (3) 
Research design and application; (4) Skills development (e.g. interview skills); (5) Data analysis 
and documentation of findings; (6) Final report (including presentations and publications); and 
(7) Action planning. We have adopted this organizing heuristic for reflecting on the first year of 
the Evaluation Team. In the next two sections, we will provide a description of the evaluation 
team and a short recount of the evaluation project as we implemented it. Following these 
sections, we will present the results of our reflections on the process of implementing the “2010- 
11 Books & Beyond Evaluation.” 

IV. Forming the Evaluation Team. 

Stevenson majored in Gender Studies and Human Biology and lived in the Global Village for 
several semesters. After working with the Documenting Team during her freshman year, she was 
recruited to serve as the leader of the Evaluation Team in her second year with Books & Beyond 
and received an AmeriCorps Service-Engagement position, allowing her to earn an AmeriCorps 
Education Award for her service. She joined the Evaluation Team based on the lead description 
provided by Books & Beyond, thinking she could use the service-learning and integrative 
approach of Human Biology in the role of Team Leader. 

Smith joined the team in his second year in Books & Beyond, after serving on the 
Fundraising Team the year before, when he had been a freshman and a resident in the Global 
Village. As a Direct Admit Scholar in Secondary Education and Spanish, he planned to study 
abroad in Spain during his junior year and was interested in finding opportunities to engage in 
research activities with faculty members. Joining the evaluation team enabled him to try out 
some new skillsets, including interacting with participants on the project. He is serving as the 
Evaluation Team Lead for the fall semester of 2012. 

Ryan officially joined the team in spring 2011, although she was significantly involved in 
the collection of video interviews in the fall due to her leadership role in the Documenting Team. 
She was a founding member of Books & Beyond, serving as a Documenting Team member 
during her first year with the project, then expanding her activities as the Documenting Team 
Lead for the academic year 2010-11, also serving as an AmeriCorps Service-Engagement Corps 
member alongside Stevenson. She became interested in the evaluation’s team work as a way for 
her to better understand the project’s scope and impact in the hopes that it would help to improve 
the project for future years. 

Samuelson joined the Books & Beyond project at its inception as a faculty advisor, when 
she was a new faculty member. During the 2008-09 and 2009-10 academic years, she worked 
intensively on the planning and editing of the first and second volumes of The World is Our 
Home, planning the first trips to TEAM in Newark, NJ, and to Kabwende Primary School in 
Kinigi, Rwanda, and engaging in fundraising that helped the project get started. Over time, 
though, Samuelson has become less involved in the daily activities of the project, but serves on 
the Advisory Board and assists primarily in matters of evaluation and curriculum development. 
Samuelson joined the Evaluation Team late in the fall semester, after the first faculty advisor to 
the team was not able to continue with the position. 
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V. The 2010-11 Books & Beyond Evaluation. 

The methodology for the Books & Beyond Evaluation reflected the needs and concerns of the 
group at large. We applied a single case participatory action research design to investigate the 
process of forming the evaluation team and conducting the evaluation. We were initially 
interested in learning why the participants had chosen to join the project and what their 
experiences had been like. Our goal was primarily to provide information that would serve to 
make the project stronger in the upcoming academic year. We provide here a brief description of 
the project and its results here for the purpose of contextualizing our reflection, as our primary 
purpose for this paper is to discuss the experience of engaging in the research. Patton has noted 
that increasing rigor in qualitative research entails getting past “lessons learned” to 
methodological rigor in service-learning evaluation, but we found that addressing this concern is 
compounded by the realities of engaging in a student-faculty partnership in a co-curricular 
service-learning context (Patton, 2012). 

A. Data Collection and Analysis. 

Early in the academic year, we reviewed methods of evaluation and developed a tentative model 
for Books & Beyond, one that would provide an effective, interactive learning experience for 
everyone involved. Consistent with the goals of the project regarding student involvement, 
development, and learning, the team decided to practice participatory evaluation, which may be 
defined as a process in which “researchers, facilitators, or professional evaluators collaborate in 
some way with individuals, groups, or communities who have a decided stake in the program, 
development project, or other entity being evaluated” (King, Cousins, & Whitmore, 2007, p. 87). 
We adopted several strategies for collecting data from project participants, including interviews, 
arts-based activities, and a written survey. 

Our first round of data collection occurred during the fall visit to Bloomington in 2010. 
Thirteen middle and high school students from TEAM Charter Schools in Newark, NJ (Rise 
Academy, grades 5-8; TEAM Academy, grades 5-8; and Newark Collegiate Academy, grades 9- 
12) visited Indiana University for a weekend in October to meet their writing mentors and get a 
taste of college life. We worked with the Documenting Team, led by Ryan to conduct short, 
videotaped interviews with each of the writing partners. Each participant was asked to why he or 
she wanted to be involved with Books & Beyond and how he or she felt about being involved. 
The arts-based activities invited participants to express their feelings about the project through 
visual art mosaics, molding clay sculptures, and devised theatre. Participants were able to 
generate pieces of artwork and creative writing throughout the weekend. The devised theatre 
activity was part of a final workshop, during which we asked the Writing Partners to create short 
performances depicting how they felt about their experiences during the weekend visit. 

We analyzed the data that we collected during the October weekend by reviewing the 
videotaped interviews and theatre activities and looking for prominent themes. We met regularly 
in Samuelson’s office to discuss the results of our content-based analysis. We noted that the 
participants valued the new mentoring relationships that they were fonning and the fun times 
they were able to share during the intensive weekend. One devised theatre piece about social 
action particularly highlighted some of the participants’ favorite parts about the project; another 
skit discussed the value of the interpersonal connections generated by the project. Additionally, 
the performances served to illuminate some of the frustrations and negative feelings that the 
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students had regarding the weekend’s activities. Several of the participants communicated their 
exhaustion from the long days and their desire to have more free time. They also displayed their 
frustration with the evaluation activities and the constant presence of the Documenting Team’s 
cameras. 

In January, the Evaluation Team accompanied the Writing Partners on their trip to 
Newark, NJ, to finish their stories and spend time with their writing mentees. During one of the 
group meetings, we presented the results of our analysis of the qualitative data we collected 
during the fall visit to the group. We also distributed the written survey, which was completed by 
fifty participants (TEAM Schools: 13; IU Global Village: 37). The first section asked 
participants to rate the severity of the problems or challenges that they experienced throughout 
the year (see Appendix). The second section, a modified version of the Civic-Minded Graduate 
Scale developing by the Center for Service-Learning at Indiana University Purdue University 
Indianapolis (Steinberg, Hatcher, & Bringle, 2011), asked participants to consider how their 
participation in the project helped them to develop a sense of connection to their schools and to 
their local and global communities. Other questions explored their interest in community 
involvement or service, their desire to pursue careers in community service, and their sense of 
confidence in their ability to make contributions to their communities. 

B. Our Results. 

The resounding theme emerging from our analysis was that students at both Indiana University 
and TEAM believed that Books & Beyond would help them to have an impact on their world. 
Students from both groups joined the project in the hope of bringing about some type of social 
change, through providing books to Rwandan schoolchildren or through increasing cross-cultural 
communication and understanding. 

Both groups of participants also offered a number of recommendations for how the 
project might be improved. In the written surveys, many participants cited time constraints as the 
main challenge or struggle that they encountered throughout their participation. However, 
additional concerns were raised throughout the process in reference to the Evaluation Team itself 
and its methods. Through the mosaics, devised theatre activity, and various points throughout the 
intensive fall weekend, participants displayed frustration with the redundancy of the Evaluation 
Team’s activities and constantly being asked, “How does this make you feel?” Taking this into 
consideration, final reflections were made about the process as a whole and what might be 
learned from this case study of youth participatory evaluation. 

We noted that most of the Global Village and TEAM Writing Partners believed that 
Books & Beyond provided them with an opportunity to “make a difference,” although the 
undergraduate Global Village participants were more likely to view the project as an opportunity 
to build their resumes. Most of the TEAM students saw the project purely as a fonn of social 
action. For these younger participants, the opportunity to engage in a social action project that 
they had control over was an especially salient factor in their decision to join the project. We also 
noted that all of the writing partners became bored and disengaged at some points during the 
writing workshops, and also during the evaluation activities. 

Both groups agreed that the biggest benefits were connecting to their global community 
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Figure 4. Perceived Benefits by IU Global Village Students of Participation in Books & Beyond 
( 2010 - 11 ). 
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Figure 5. Perceived Benefits by TEAM Students of Participation in Books & Beyond (2010-11). 


and servicing others, and both groups had the highest percentage of students agreeing with two 
statements: “My Books & Beyond experiences have helped me to develop a sincere desire to be 
of service to others” and “Participating in Books & Beyond increased my connection to my 
global community” (see Figures 4 & 5). All of the TEAM Schools students agreed or strongly 
agreed that the project had helped to increase their desire to be of service to others, whereas 76% 
of Indiana University students agreed or strongly agreed. The Indiana University students 
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showed strong agreement that Books & Beyond had increased their connection with the global 
community; 86% agreed or strongly agreed (Figure 4). TEAM students were more likely to agree 
that their Books & Beyond experiences had increased their connection to their global 
community, increased their desire to address community issues and increased their confidence in 
their ability to be involved in addressing community issues (Figure 5). TEAM students show a 
stronger benefit of serving others in their communities and globally, while the Indiana University 
students felt they most benefited in the area of global connections. 

The Indiana University students seemed primarily concerned with the amount of time 
devoted to writing and training; thirty percent indicated that time constraints were a “big 
problem” or a “really big problem” (see Figure 6). In the open-ended questions, some stated that 
they would prefer more time with their writing mentees devoted to socializing or working on 
their stories and less time spent in meetings. Some of their comments included: 

1. “More quality time with my writing partner.” 

2. “We could have more social and work time.” 

3. “You probably should just cut back on the lectures.” 

4. “Some changes that I think should be made are adding more times to simply be with your 
writing partner.” 

5. “Many GV students express displeasure often times with being documented. More effort 
in explaining the crucial roles we all play, even the documenting team.” 

6. “Hold more Rwandan culture nights.” 

The TEAM students were also concerned about time constraints. Twenty-three percent 
indicated that time constraints were a “big problem” or a “really big problem.” The TEAM 
students were more likely to indicate that the Skype sessions, the Saturday meetings with their 
GV writing partners, and the time needed to maintain communication with their GV writing 
partners were also significant challenges (see Figure 7). These students desired additional time to 
Skype with their partners and suggested holding writing and mentoring sessions during the week 
instead of the weekend. Some of their comments in the open-ended questions on the survey 
included the following: 

1. “I think that you should make what you want accomplished clearer.” 

2. “Next we could have more Skype sessions and more time to communicate.” 

3. “I've really enjoyed the program. Seems like we could do a little more with maybe having 
Skype sessions with students from Rwanda/Kenya.” 

4. “Next year we should have more classes that help us learn about Rwandan culture.” 
During the devised theater event, we observed both the Indiana University students and 

the TEAM students speaking optimistically about the power of the project to break down 
barriers. The students expressed a desire to have more free time with their writing partners so 
they could get to know one another without having structured activities. They were highly 
motivated to accomplish the task of completing their stories on deadline. 

Reviewing the results of the 2010-11 Books & Beyond Evaluation helped to shed light 
on why students chose to be involved and whether they thought that project helped them to be 
community-minded. Most of the challenges that the students highlighted were addressed in 
action steps that the Books & Beyond Leadership Team incorporated into subsequent years. 
Many respondents expressed a desire for more one-on-one time to work with and get to know 
their writing partners. Respondents also cited an interest in learning more about the nation of 
Rwanda. In response to these requests, 2011-12 and 2012-13 saw increased opportunities during 
the October and January Weekends for writing partners to spend time with one another and work 
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on their stories. With the Evaluation Team as a framework, Books & Beyond hired a student 
intern in 2011-12 with the primary task of developing a strategic plan for the project that 
included both short-term and long-term goals. Most recently, the 2012-13 academic year saw the 
creation of a Rwanda Culture Team within Books & Beyond, which hosts a series of informative 
events each semester at the Global Village Living-Learning Center addressing the history and 
culture of Rwanda. 


| - 
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TEAM students GV students theme conferencing) meetings English learner levels Rwandan culture 


Figure 6. Challenges Experienced by IU Global Village Students in Books & Beyond (2010-11). 
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Figure 7. Challenges Experienced by TEAM Students in Books & Beyond (2010-11). 
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D. Action Steps. 

The action steps phase covers the presentation of findings and their application to the project. We 
made a full presentation to the Books & Beyond Leadership Team at the end of the year. As a 
result of our findings, the project made some significant changes in the subsequent year. General 
awareness of and appreciation for evaluation of the project increased among leadership team 
members, and methods of evaluation were altered to better suit the project’s needs and to avoid 
overwhelming student participants with redundant questioning. The Public Relations Team was 
added to the project and began publishing the Amakuru 2 newsletter. Rwandan culture nights 
became a regular activity for Indiana University students engaged in the project. And, finally, 
students were given more “down time” together during the fall and spring visits to encourage 
writing partners to get to know each another better. 

We presented our results of our study at several forums, including the IU Women in 
Science Research Day Conference, the IU Hutton Honors College Undergraduate Research 
Symposium, and the Indiana Campus Compact Service Engagement Summit. 

VI. Reflecting on the 2010-11 Books & Beyond Evaluation. 

Although the study resulted in the collection of a wealth of data regarding participants’ thoughts, 
feelings, and preferences, the process of conducting the youth participatory evaluation was 
shown to be just as important as the evaluation itself (Voakes, 2003). It gave us an opportunity to 
practice and develop a new set of skills in research collection and analysis, evaluation 
implementation, and non-profit program support. This type of evaluation gave us the tools we 
needed to develop and validate knowledge while also addressing the development of Books & 
Beyond. 

During the spring semester, we realized that the process of engaging in the Books & 
Beyond Evaluation was just as interesting as the results of the evaluation. We agreed that after 
we had completed the evaluation, we would reflect on the process and note what we had learned 
about the process of youth participatory evaluation as well as areas for improvement for future 
evaluations. During the summer and early fall of the following academic year (2011-12), each of 
the team members met individually with Samuelson to discuss their reflections on the seven 
distinct steps in designing and implementing a youth participatory evaluation program (Tagor & 
Cuellar, 2008): (1) Project planning and start-up; (2) Team-building (including critical thinking, 
skills development); (3) Research design and application; (4) Skills development (e.g. interview 
skills); (5) Data analysis and documentation of findings; (6) Final report (including presentations 
and publications); and (7) Action planning. In the remainder of this section, we will address our 
reflections on each of these steps in turn. 

A. Reflections on Project Planning and Start-Up. 

In the project planning and start-up phase, the Evaluation Team was assembled and started its 
planning meetings. Smith was hired in August by the Books & Beyond Leadership team, and 
helped to recruit Stevenson. Samuelson was involved in the process, but she did not step in as the 
faculty advisor until November, when the first advisor assigned to the team was no longer able to 
continue. She was aware of how the team was developing and how the planning was proceeding, 


2 “Amakuru” is a Kinyarwanda greeting that means “Hello” or “How’s it going?” 
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but was not directly involved. In some ways, the planning phase was the most difficult because 
of the decision-making that needed to go into decision what questions to ask, what types of data 
to collect, and how to analyze it. Each of the team members also had to complete the training 
required to getting approval from the university’s Institutional Review Board, a process that 
proved very time-consuming. Looking back at this period, Smith felt that the early planning 
phase of the project involved too many planning meetings in which ideas got discussed, but 
decisions didn’t get made: 

I prefer working with a hierarchy in a team; no clear chain of command. It wasn’t really 
clear who was making decision. [..] The way I work as a learner, just give me two or 
three different methods and a plan of attack (Interview, August 3, 2011). 

He also felt that it would have been helpful to read related research early in the year. Last 
minute changes to the plans were frustrating for all of the team members. 

B. Reflections on Team-Building. 

When Ryan joined the team in the spring, the team spent several weekends together researching 
and preparing for their poster presentations, and they felt that they worked well together during 
this process. At this point, the team began to see the value in spending time completing work 
together as a means of building camaraderie and improving the quality of their work. This 
seemed to be more motivating and engaging than the multiple, long, planning sessions held 
during the first semester. The students also bonded over the shared sense of challenge and 
urgency presented by their goals to present in on campus conferences during the spring semester. 
The need to complete a presentation and poster display brought the group together in working 
towards a common goal. 

C. Reflections on Research Design and Application. 

During the design and application phase, we observed that every little action that the writing 
partners perfonned counted, because their activities together—enjoying free time, eating lunch, 
falling asleep during lessons, speaking in groups during activities, etc., helped to strengthen their 
mentoring relationships. Because of this, we emphasized the importance of noting even the 
“bad” or “embarrassing” things that happened. We also discovered the importance of allowing 
for enough time to analyze all of the data. This time-consuming process worked best when we 
planned to do it over the winter break. 

A few details about the leadership of the team arose at this stage in the reflection. Smith 
expressed a preference for greater control by faculty members, and would have preferred more 
top-down instructions on what to do and how to do it. Stevenson wanted to leam more about 
different techniques for interviewing, and wondered how the outcomes might have differed 
based on different types of questions. Stevenson wanted to collect more statistical data at the 
beginning of the year, and not just at the end of the project. 

Stevenson also felt that although it was important to have multiple types of data, she 
would have preferred skipping some of the arts-based activities: 

Our first set of data collection would have been more helpful if we had had less 
“elementary art.” I wish that it had been less scattered, like having everyone just sit down 
and write a poem or something, instead of waiting until someone tried to sit down and do 
their homework. It was really difficult to get a response that was more than just, “I’m 
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exhausted.” [...] I do like the art form. I think it was really helpful, but a lot of the 
supplies were really hard to carry around. And it was hard to keep the pieces of art intact. 
It was really difficult (Interview, September 16, 2011). 

D. Reflections on Skills Development. 

Smith noted that being involved in the project helped him to rethink his plans for a future career: 

I feel that I developed at least some sort of foundation of research skills. I developed 
more of an appreciation for reading research that is out there than I had ever done for any 
class, maybe because this was research that was relevant to what I was doing. I started to 
really notice things in a research paper, like, ‘oh they’re citing this author. I should go 
back and look at the research this person has done.’ Or like, ‘what is this person saying 
versus what this individual is saying?’ That came more into play in the spring when we 
were starting to do the poster presentations around campus. I enjoyed that because I never 
really made any sort of contribution to IU in that way, to attend a conference. It was a 
really cool experience for me. And I vastly improved my presentational skills, which is 
always a work in progress for everyone. Getting involved in evaluation got me thinking 
more and more about working in academia, which is something I hadn’t really considered 
prior to this past spring (Interview, August 3, 2011). 

Ryan noted that many of the skills that she had acquired during her involvement with 
Books & Beyond had little to do with her academic classes. She reported developing a greater 
appreciation for reading research, more than she had gained in any of her classes. The 
experiences were valuable, but she had difficulty translating this into a resume or a cover letter: 

A lot of times, I’ll write in cover letters, or something like, ‘I’ve learned more or just as 
much through doing this project as I have in the classroom here.’ Not only is it awesome 
because I’m working on this completely different side of myself than what is my major 
and my minor but it’s also just great because I’m learning though my own initiative, too, 
I learned all of that awesome stuff about Rwanda and about evaluation and really cool 
stuff last year, and I never would have gotten that in a classroom, because I never would 
have taken a class on it, and there isn’t one on it. And it was just great. I wish that more 
people could have that experience. Or I wish that it would count for something 
(Interview, September 15, 2011). 

Stevenson wished for more training in interview techniques: 

I wish we would have learned more about different ways of interviewing people. I would 
have liked to see different ways students responded depending on the way that we 
phrased the questions (Interview, September 16, 2011). 

E. Reflections on Final Reports and Presentations. 

The team agreed that presenting the data at academic conferences and to Books & Beyond 
members helped to promote the project. Their presentations brought further benefits by helping 
them to see the importance of what they were doing and by motivating them to keep working. At 
the end of the 2010-11 Books & Beyond final presentation to the Leadership Team, the 
Evaluation Team members agreed that writing an article on their results and their experiences 
would be a valuable next step, and they drafted a report on the results that formed the basis for 
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the first part of this paper. Stevenson found this step very enjoyable: 

The poster was basically a condensed version of the paper. Because there were three of us 
working on it, we used the time really well and made sure that we didn’t procrastinate on 
it. We spent an entire day together just looking for resources. The next weekend after that, 
we made the poster from the paper. So it was just a lot of finding and then putting 
together the stuff and revising. So it worked out really well (Interview, September 16, 
2011 ). 

The presentations helped us to demonstrate the importance of our work to other Books & 
Beyond participants. Smith shared that “[At the Books & Beyond leadership year-end meeting], 
someone came up to me and said, ‘you guys have been kind of getting on my nerves all year, but 
now I finally understand why you have done all of this,’ which is really a gratifying moment.” 

F. Reflections on Action Steps. 

We compiled many of these observations in the 2011 Books & Beyond Evaluation Team 
Leadership Legacy Manual, which we made available to the 2011-12 Evaluation Team. Several 
of our suggestions for future evaluation teams appear in the Appendix. Our team disbanded as 
Smith left for Spain for a study abroad year, Stevenson focused on completing her studies, and 
Ryan took on other duties with the Books & Beyond Leadership Team. Despite the team 
disbanding, all three authors maintained a sense of involvement, to varying degrees, in ensuring 
the project’s future success and supporting future evaluation teams. 

VII. Implications and Future Directions. 

Our rationale for this qualitative study was to examine the value of YPE for enhancing learning 
opportunities for participants and for producing more relevant results for the Books & Beyond 
project. Our purpose was to examine our experiences and learn from them. Conducting a youth 
participatory evaluation of Books & Beyond helped us to generate a great wealth of participant 
insights about the project, providing us with knowledge about what Books & Beyond 
participants get from their involvement as well as how the project might be improved for the 
future. We set out to collect data on the success of Books & Beyond, report our findings to the 
participants and to the broader community, and finally, to reflect on the process of our evaluation 
project and ways that it could be improved. 

The Books & Beyond Evaluation Team has demonstrated how engaging project 
participants in evaluation has produced an atmosphere conducive to continual project 
improvement and adjustment for the sake of the greater goal of bringing about social change in 
the world. In future years, we hope that Books & Beyond will be able to engage the TEAM 
students more actively in the evaluation process. Furthermore, although we elected not to collect 
evaluation data from the Rwandan students at Kabwende Primary School, we would like to see 
this step occur, as the Rwandan students have participated in the project by writing and 
illustrating their stories and by receiving copies of the book for their personal use. The goal of 
ensuring that others benefit from our experiences extends not only to the future participants of 
Books & Beyond, but to other student-faculty teams who may desire to engage in a collaborative 
evaluation of their co-curricular service activities. 
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Appendices 

Appendix 1. Books & Beyond Satisfaction Survey 

We want to hear from you about your experience with Books & Beyond so we can make 
improvements for next year. 

1. Part One 

la. I am from [please check your school]: (Indiana University or TEAM) 

lb. My gender is (Female or Male). 

2. What were your roles in Books & Beyond this year? [check all that apply] (Writing 
Partner, Collaborator, Pen pal, Fundraiser, Documenter, Evaluator, or other) 

3. What was your MAJOR role in Books & Beyond this year? (Writing Partner, Collaborator, 
Pen pal, Fundraiser, Documenter, Evaluator, or other) 

4. For the role that you checked in #3, what are some areas that you would like to do 
differently, if you were to join Books & Beyond again next year? 

5. Approximately how much time did you devote to the project (in all your roles)? (1-2 hour 
per week; 3-4 hours per week; 5-6 hours per week or Other) 

6. We know you faced many challenges in completing the project. Please rate the following: 

(Very small problem. Small problem, Not a problem, Big problem, Really big problem, Not 
applicable to me) 

a. time constraints 

b. communicating with TEAM students 

c. communicating with GV students 

d. uninteresting story theme 

e. using Skype (online conferencing) 

f. using MS Publisher 

g. attending Saturday meetings 

h. writing for different English learner levels 

i. understanding Rwandan culture 

j. Other (please specify) 

7. Please suggest some changes we could make for next year that would address these 
challenges. 
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8. Why did you decide to be a part of Books & Beyond? 

II. Part Two 

(Very small problem. Small problem. Not a problem, Big problem, Really big problem, Not 

applicable to me) 

1. Participating in Books & Beyond increased my connection to my school (IU, TEAM 
Academy, RISE Academy or Newark Collegiate Academy). 

2. Participating in Books & Beyond increased my connection to my local community (Newark, 
NJ, or Bloomington, IN). 

3. Participating in Books & Beyond increased my connection to my global community. 

4. My Books & Beyond experiences have helped me to realize that I like to be involved in 
addressing community issues. 

5. My Books & Beyond experiences have helped me develop a sincere desire to be of service to 
others. 

6. Based on my experiences with Books & Beyond, I would say that the main purpose of work 
is to improve society through my career. 

7. My experiences with Books & Beyond have increased my confidence that I can contribute to 
improving life in my community. 

8. FOR TEAM STUDENTS ONLY: Participating in Books & Beyond increased my connection 
across schools (TEAM Academy, RISE Academy and Newark Collegiate Academy). 

Appendix 2. Suggestions from the 2011 Books & Beyond Evaluation Team Leadership 

Legacy Manual 

1. Have a casual meeting first where everyone just gets to know one another. 

2. Make sure to present yourself as one of the team and not just as the leader. This is a small 
team, and the other members will be doing almost as much work as you do for Books & 
Beyond. 

3. After the first meeting, establish a weekly meet-up with your teammates that would start the 
next week. Be sure to contact everyone about the meetings a few days in advance and the day 
of the meeting so that nobody forgets. 

4. Create an agenda for what you want to talk about at the weekly meetings. 

5. Be sure to take notes at the meetings so that you don’t forget what was discussed, what is 
coming up, and what needs to be addressed in the future meetings. 

6. Working with faculty will give you access to advice that will help you get through any 
challenges or obstacles. Be sure to include the faculty in all scheduled meetings and updates, 
even if faculty can’t make every meeting or event. 

7. Brainstorm ideas for collecting data before addressing your team. It’s always good to get the 
ideas going with your own first, and then the rest of the team will possibly start 
brainstonning along with you. 

8. Once ideas get flowing, take notes of everyone’s ideas. Be sure to keep the ideas available 
until an absolute final decision is made on how you and your team will collect data. 

9. Work with your team to choose a few methods on how to collect data. Don’t collect data in 
only one format; this will not be helpful because the writing partners and other students in the 
project respond to different ways of asking “How do you feel?” 

10. Make sure you give everyone enough time to analyze all of the data so that in a near meeting, 
your team can compare notes. Take notes on major themes you notice and give examples of 
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them. Make note of any exceptions as well. 

11. After discussing what you and your team found in the data, begin making a compilation of your 
findings. 
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